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1  -  Summary 


Formal  analysis  of  computer  systems  ultimately  relies  upon  aeeurate  mathematieal  models.  When 
systems  are  software  based,  models  ean  be  ereated  based  on  software  semantics  and  the  underlying 
platform.  When  systems  involve  the  physieal  world,  a  model  identification  step  may  be  performed. 

A  deviation  of  system  behavior  from  the  model  indieates  that  behaviors  may  oeeur  whieh  were  not 
antieipated  during  system  design  and  analysis.  Sueh  deviations  may  oeeur  if  the  model 
identifieation  step  was  done  ineorrectly,  if  simplifications  (such  as  linearization  of  nonlinear 
components)  are  oversimplifications  of  actual  behavior,  if  the  environment  at  design  time  varies 
from  the  environment  at  deployment,  or  if  wear  and  tear  of  eomponents  ehanges  their  behavior.  In 
software,  model  mismatehes  may  be  indieative  of  seeurity  flaws  in  the  implementation  being 
exploited  to  produee  unintended  behavior.  In  any  ease,  the  results  of  any  earlier  formal  analysis 
do  not  apply  when  the  model  is  not  an  accurate  representation  of  the  system. 

This  researeh  effort  investigated  fundamental  teehniques  to  provide  perpetual  model  validation, 
where  design-time  models  are  validated  eontinuously  during  runtime.  The  researeh  eonsidered  two 
fronts:  validation  of  the  software  model,  and  validation  of  the  model  of  the  system  interacting  with 
the  physieal  world.  Since  the  field  of  runtime  verification  has  extensively  focused  on  the  ehallenge 
of  runtime  model  validation  in  software  using  direet  models,  the  approaeh  employed  here  instead 
eonsidered  using  indireet  models  of  software  exeeution,  for  example  memory  aeeess  patterns,  to 
check  for  security  intrusions.  Additional  research  was  performed  to  taekle  the  essential  problem 
of  model  validation  for  systems  which  contain  interactions  with  the  physical  world  using  hybrid 
automata  models.  Perpetual  model  validation  will  ensure  that  the  aetual  behavior  of  the  system 
conforms  to  the  analysis  model,  raising  the  level  of  eonfidence  that  can  be  placed  in  the  results  of 
formal  analysis. 
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2  -  Introduction 


A  growing  percentage  of  systems  across  a  variety  of  domains  are  comprised  of  Cyber-physical 
systems  (CPS).  Often  these  systems  belong  to  such  domains  as  vehicles,  power  plants,  and  medical 
devices  whose  security  and  reliability  are  critical  to  the  safety  and  well-being  of  their  users.  In  the 
past  such  systems  were  generally  considered  to  be  relatively  safe  from  malicious  adversaries 
because  they  were  isolated  from  their  environments  and  running  specialized  software  on  unique 
hardware  platforms.  However,  this  is  becoming  less  and  less  the  case,  especially  with  regard  to 
connectivity,  where  Cyber-physical  systems  are  increasingly  interacting  with  their  environment 
through  short  and  long  range  wireless  media. 

2.1  -  The  Challenge  eor  the  Air  Force 

The  mission  of  the  Air  Force  is  to  “fly,  fight  and  win  in  air,  space  and  cyberspace”.  Software  is  a 
key  contributor  to  meeting  the  requirements  necessary  to  fulfill  the  Air  Force  mission.  “Simply 
stated,  absent  secure  and  resilient  software  at  the  core  of  our  cyber  defenses,  the  nation’s  critical 
infrastructure  is  at  risk”  [1].  Software  controls  an  increasing  number  of  mission  critical  systems 
ranging  from  planning  to  weapon  systems.  Since  software  failures  can  cause  extensive  damage 
resulting  in  loss  of  productivity,  loss  of  mission  capability,  loss  of  assets,  and  loss  of  life,  it  is 
important  that  these  systems  are  developed  and  verified  to  be  correct.  Software  for  mission  assured 
systems  should  consist  of  several  attributes  including  but  not  limited  to  correctness,  security, 
safety,  resilience,  availability,  performance  and  reliability.  The  correctness  of  software  can  be 
increased  with  a  design  environment  that  allows  for  the  modeling,  early  analysis,  and  synthesis  of 
software. 

General  purpose  software  development  has  demonstrated  that,  with  almost  certainty,  software  bugs 
will  be  present  in  the  resultant  code.  For  this  reason,  formal  methods  must  play  a  larger  role  in 
future  Air  Force  software  systems.  Safety  verification,  building- in  security  from  the  earliest  design 
steps,  and  other  early  analysis  approaches  all  require  models  to  be  made  which  capture  the 
expected  behavior  of  the  system. 

Formal  analysis  of  computer  systems  ultimately  relies  upon  accurate  mathematical  models.  When 
systems  are  software  based,  models  can  be  created  based  on  software  semantics  and  the  underlying 
platform.  When  systems  involve  the  physical  world,  a  model  identification  step  may  be  performed. 
A  deviation  of  system  behavior  from  the  model  indicates  that  behaviors  may  occur  which  were  not 
anticipated  during  system  design  and  analysis.  Such  deviations  may  occur  if  the  model 
identification  step  was  done  incorrectly,  if  simplifications  (such  as  linearization  of  nonlinear 
components)  are  oversimplifications  of  actual  behavior,  if  the  environment  at  design  time  varies 
from  the  environment  at  deployment,  or  if  wear  and  tear  of  components  changes  their  behavior.  In 
software,  model  mismatches  may  be  indicative  of  security  flaws  in  the  implementation  being 
exploited  to  produce  unintended  behavior.  In  any  case,  the  results  of  any  earlier  formal  analysis 
do  not  apply  when  the  model  is  not  an  accurate  representation  of  the  system. 

Autonomy  in  systems,  provides  the  Big  Challenge  as  we  see  it,  and  reason  why  formal  methods 
must  have  an  end-to-end  role  in  our  system  development,  beginning  at  the  earliest  of  system 
incarnation.  Figure  1  shows  Cyber  Resilience  along  the  x-axis  and  Trust  on  the  y-axis,  with  the 
goal  being  a  system  which  is  highly  trusted  and  resilient  against  even  zero-day  attacks.  Let  us 
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define  that  Trust  represents  the  users’  belief  in  the  reliability  and  effectiveness  of  the  system  as 
measured  by  its  correctness  and  security.  In  other  words,  Trust  and  Cyber  Resilience  are  the  ability 
of  the  CPS  to  continue  to  operate  and  maintain  mission  essential  functions  while  coping  with  on¬ 
going  cyber-attacks  or  system  failures.  To  achieve  these  goals,  we  have  been  pursuing 
development  of  techniques,  methodologies  and  tools  to  enable  trust  and  resilience  (as  measured 
by  correctness,  security,  reliability,  predictability,  and  survivability)  and  migrate  the  analysis  from 
execution  (testing  and  monitoring)  to  design  (correct  and  formal/security  specifications)  and 
development  (composition  and  auto-generation). 


Complete 


Partial 


Untrusted 


No  Attacks  Failures  Attacks  Zero-Day 

or  failures 


Resilience 


Figure  1:  Towards  Trusted  and  Resilient  Systems. 


2.2  -  Objective  oe  this  eeeort 

Perpetual  model  validation  enhances  and  complements  these  formal  method  techniques,  by 
checking  that  the  expected  model  reflects,  as  far  as  one  can  tell,  the  behavior  of  the  deployed 
system.  If  a  violation  is  detected,  it  does  not  necessarily  mean  that  the  formal  properties  are  wrong 
and  will  be  violated,  only  that  their  proof  is  inadmissible.  This  in  itself  is  extremely  valuable 
information,  and  inferring  it  before  stressing  the  system  can  prevent  unexpected  violations  of 
safety  and  security.  This  research  project  aims  to  increase  confidence  in  the  correctness  of  the 
models,  and  therefore  increase  confidence  in  the  correctness  of  the  resultant  formal  guarantees. 

This  research  effort  investigated  fundamental  techniques  to  provide  perpetual  model  validation, 
where  design-time  models  are  validated  continuously  during  runtime.  The  research  considered  two 
fronts:  validation  of  the  software  model,  and  validation  of  the  model  of  the  system  interacting  with 
the  physical  world.  Since  the  field  of  runtime  verification  has  extensively  focused  on  the  challenge 
of  runtime  model  validation  in  software  using  direct  models,  the  approach  employed  instead 
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considered  using  indirect  models  of  software  execution,  for  example  memory  access  patterns,  to 
check  for  security  intrusions.  Additional  research  was  performed  to  tackle  essential  problems  of 
model  validation  for  systems  which  contain  interactions  with  the  physical  world,  using  hybrid 
automata  models.  Perpetual  model  validation  seeks  to  ensure  that  the  actual  behavior  of  the  system 
conforms  to  the  analysis  model,  raising  the  level  of  confidence  that  can  be  placed  in  the  results  of 
formal  analysis. 

Models  of  computer  systems  enable  a  myriad  of  formal  analysis  and  verified  design  approaches. 
Sound  application  of  formal  methods  can  guarantee  with  certainty  properties  regarding  the 
behavior  of  models  of  software  systems  and  the  way  in  which  they  will  interact  with  models  of 
their  environment.  However,  any  formal  guarantees  proven  about  the  system  are  inapplicable  when 
the  model  used  for  the  proofs  does  not  correspond  to  the  deployed  system.  This  research  effort, 
therefore,  sought  to  investigate  fundamental  techniques  to  increase  the  confidence  that  a  system’s 
analysis-time  model  corresponds  to  the  deployed  system. 

Significant  research  efforts  have  been  underway  at  the  Air  Force  Research  Laboratory  (AFRL)  to 
encourage  the  use  of  formal  methods  and  model-based  design  in  software  development.  The 
“Correct-by-Construction  Software  for  Embedded  Multi-core  Systems”  (CxC  SEMS)  is  one  effort 
where  provably  correct  models  of  reactive  systems  are  used  to  generate  multicore  implementation 
code,  and  then  another  step  will  be  used  to  prove  the  generated  code  corresponds  to  the  initial 
model.  This  second  step  is  essential  since  a  divergence  between  code  and  model  invalidates  any 
guarantees,  which  is  also  the  motivation  behind  perpetual  model  validation.  Another  effort  at 
AERE  is  the  Office  of  Secretary  of  Defense  (OSD)  sponsored,  “Techniques  and  Tools  for 
Trustworthy  Composition  of  Pre-Designed  Embedded  Software  Components”.  In  this  work, 
software  components’  indirect  assumptions,  for  example  timeliness,  are  exported  along  with  the 
components  to  enable  more  dependable  integration.  Perpetual  model  validation  could  be 
compatible  with  this  work  by,  for  example,  monitoring  these  indirect  assumptions  at  runtime  to 
validate  that  they  conform  to  the  specification.  Easily,  the  Air  Eorce  Office  of  Scientific  Research 
(AEOSR)  Eaboratory  Research  Initiation  Request  (ERIR)  “Design  and  Analysis  of  Trustworthy 
Software”  is  making  use  of  domain-specific  models  to  perform  upfront  analysis  of  large  software 
systems.  After  early  analysis  of  models,  trusted  code  generation  creates  software  which  conforms 
to  the  models,  and  artifacts  are  generated  to  help  with  testing  and  deployment.  These  artifacts  could 
potentially  include  elements  of  perpetual  model  validation  to  check  that  non-software  components 
conform  to  their  analysis  models.  Eor  example,  software  timeliness  ultimately  depends  not  only 
on  the  software  itself,  but  also  on  a  host  of  other  factors  including  the  underlying  CPU,  memory 
hierarchy,  cache  interference  from  other  programs  running  in  the  system,  the  effectiveness  of 
speculative  prefetch  and  branch  predictors,  and  the  effects  of  memory- access  reordering  in  the 
DRAM  controller.  Perpetual  model  validation  can  monitor  factors  like  timeliness,  to  make  sure 
the  underlying  system  meets  requirements  from  the  models. 

This  research  effort  contributes  toward  the  science  needed  to  build  safe  and  secure  systems.  It  is 
important  for  the  success  of  the  Air  Eorce  to  overcome  current  approaches  which  consider  security 
in  the  late  stages  of  the  development  of  a  system,  and  then  with  formulations  of  security  incapable 
of  providing  comprehensive  guarantees.  The  result  is  a  proliferation  of  attacks.  One  of  the 
Essential  Eocus  Areas  for  the  Air  Eorce  is  Cyber  Resilience,  whereby  systems  would  be  able  to 
withstand  attacks.  A  superior  approach  is  to  prevent  many  of  those  attacks.  Research  into  perpetual 
model  validation  contributes  toward  that  goal. 
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2.3  -  Review  oe  State-oe-the-Art 


Runtime  verification  [2,  3]  methods  employ  monitors  which  check  a  program's  behavior  against  a 
model  of  the  expected  behavior.  For  example,  an  API  may  require  functions  are  called  in  a 
particular  order,  or  software  can  require  a  resource  is  acquired  and  released  in  a  particular  function. 
In  classic  runtime  verification,  monitor  code  is  inserted  into  the  original  program,  and  properties 
are  checked  at  runtime.  This  classical  checking  approach  uses  direct  models.  Direct  model 
validation  need  not  be  limited  to  the  CPU;  for  example  one  runtime  verification  approach  monitors 
PCI  bus  transactions  to  make  sure  that  registers  being  read  and  written  on  an  FO  peripheral  satisfy 
a  device  interface  model  [4] . 

In  perpetual  model  validation,  the  proposed  research  will  instead  focus  on  indirect  models,  which 
capture  aspects  that  arise  as  a  result  of  the  implementation  (side  channels).  While  side  channels 
are  typically  mentioned  in  the  context  of  attacking  cryptography  systems  by  measuring  timing  [5] 
or  power  fluctuations  [6],  they  are  employed  here  instead  as  a  defense  mechanism  against  code 
injection.  For  example,  the  proposed  memory  access  patterns  are  not  something  directly  specified 
by  the  software  engineer,  but  instead  arise  as  byproducts  of  the  implementation.  In  earlier  work, 
this  technique  has  been  shown  as  sound,  by  monitoring  timing  channels  of  real-time  systems  to 
detect  intrusions  [7].  Other  work  has  successfully  shown  that  monitoring  things  like  chains  of 
system  calls  can  differentiate  normal  system  behavior  from  that  of  malware  [8]. 

In  terms  of  systems  containing  physical  aspects,  formal  control  theoretic  approaches  exist  for 
model-based  fault  diagnosis  for  purely  continuous  systems  [9,  10].  This  was  generally  done  for 
analytical  redundancy,  that  is,  to  use  software  to  detect  when  a  sensor  or  actuator  has  failed.  In  the 
proposed  research,  however,  models  will  be  used  of  combined  software  /  hardware  systems 
represented  as  hybrid  automata  which  also  include  discrete  system  states  not  considered  with  the 
control  theoretic  approaches. 

Model  checking  frameworks  such  as  Maude  [11]  can  be  used  to  compute  reachability  and  check 
safety  specifications  for  deterministic  and  nondeterministic  discrete  systems,  without  continuous 
states.  Extensions  of  Maude  such  as  Real-Time  Maude  [12]  and  HI-Maude  [13]  allow  continuous 
states  to  be  expressed  and  analyzed  within  the  Maude  engine,  although  complete  analysis  is  limited 
to  decidable  classes  of  hybrid  automata  (with  timed  or  rectangular  dynamics).  Systems  with  more 
complicated,  nonlinear  dynamics,  can  be  sampled  through  time,  although  this  strategy  is  generally 
not  sound  (it  may  “verify”  a  model  which  actually  can  reach  error  states).  In  the  proposed  approach 
and  research,  the  reachable  set  of  states  for  nonlinear  hybrid  automata  is  over  approximated, 
preserving  soundness  at  the  cost  of  potential  false  positives. 

Other  researchers  have  also  recently  considered  performing  online  model  checking  for  hybrid 
systems  [14,  15].  The  proposed  approach  was  used  to  verify  a  parameterized  system  by 
constructing  a  new  hybrid  automaton  online  for  the  current  instance  of  the  problem,  and  the 
purpose  was  to  drive  a  supervisory  controller.  This  approach,  therefore,  was  susceptible  to  errors 
in  the  model,  just  as  offline  analysis  approaches.  The  proposed  research  instead  strives  to  validate 
the  system  model,  so  it  is  compatible  with  this  earlier  work.  From  a  theoretical  standpoint,  the 
computation  considered  was  one  of  safety  (there  was  an  explicit  unsafe  state)  rather  than 
reachability,  and  only  linear  hybrid  systems  were  allowed.  Furthermore,  the  reachability  algorithm 
was  not  considered  in  the  research  (instead  it  was  treated  like  a  black  box),  so  its  worst-case 
performance  was  not  established.  In  the  proposed  research,  in  contrast,  systems  will  be  considered 
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with  more  general,  nonlinear  dynamics,  and  a  deadline-aware  algorithm  to  do  the  online 
reachability  computation  will  be  investigated  for  the  purpose  of  perpetual  model  validation. 

Another  approach  which  considers  online  hybrid  systems  model  checking  does  it  because  a 
medical  device  safeguard  system  is  considered  [16]  where  the  human  body  is  part  of  the  system. 
They  cannot,  therefore,  construct  an  offline  model  with  enough  accuracy  and  instead,  use  online 
analysis  to  infer  a  model  from  runtime  data.  Then,  based  on  the  constructed  model  they  drive 
supervisory  control  logic.  Rather  than  providing  timeliness  guarantees  for  the  online  checks,  the 
authors  measure  how  often  the  online  check  fails  to  meet  deadlines  and  reason  about  the  possible 
consequences.  The  accuracy  of  the  model  is  considered  (since  it  was  constructed  online),  which  is 
similar  in  spirit  to  the  idea  of  perpetual  model  validation.  Here,  a  probabilistic  argument  is  made 
about  how  parameters  in  the  model  may  be  wrong  where,  again,  the  consequences  of  the  model 
inaccuracies  are  reasoned  about.  It  is  argued  that  medical  safeguards  commonly  strive  to  reduce 
the  chance  of  unsafe  states,  so  a  probabilistic  guarantee  is  still  useful  for  the  considered  system. 
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3  -  Methods,  Assumptions  and  Procedures 


Cyber-physical  systems  are  increasingly  interacting  with  their  environment  through  short  and  long 
range  wireless  media.  Our  research  focuses  on  checking  the  accuracy  of  continuous  models  at 
runtime.  Applying  formal  analysis  techniques  to  CPS  in  such  a  way  requires  accurate  mathematical 
models  and  can  allow  for  the  detection  of  attacks  or  degraded  performance.  Our  approach  to  this 
is  to  validate  the  software  and  that  software’s  interaction  with  its  environment  through  hybrid 
automata  based  modeling. 

The  “big  picture”  of  the  proposed  research:  as  described  in  the  task  narrative,  this  research  can 
address  the  crucial  challenge  of  increasing  the  trustworthiness  of  formal  approaches  to  system 
design.  The  research  directions  proposed  are  motivated  by  experience  with  designing  predictable 
systems  and  formal  design  methods,  summarized  as  follows: 

•  Software  implementations  contain  deterministic  side  channels  which  can  be  used  as 
indirect  models  of  their  execution.  By  measuring  last  level  cache  misses  on  SPEC2006 
benchmarks,  one  can  see  clear  differences  in  memory  access  patterns  among  different 
benchmark  programs. 

•  Memory  access  patterns  can  be  modified  without  significantly  affecting  observable 
program  behavior.  By  prefetching  memory  at  the  application  level,  memory  access  can  be 
clustered  into  specific  regions  in  the  code.  For  several  benchmarks  in  the  MiBench 
benchmark  suite  [17],  this  clustering  did  not  result  in  increased  execution  time  or  total 
memory  usage. 

•  Current  hybrid  automata  analysis  techniques  allow  only  limited  analysis  of  physical 
systems  with  nonlinear  dynamics.  By  improving  on  earlier-developed  approaches,  it  is 
possible  to  analyze  nonlinear  hybrid  automata  in  a  small  number  of  dimensions,  or 
alternatively  over  a  short  time  horizon. 

The  ultimate  vision  is  a  system  where  formal  methods  are  used  up-front,  at  system  design  time,  to 
provide  guarantees  of  safety  and  security.  Perpetual  model  validation  is  employed  at  runtime  on 
the  same  models  that  were  used  to  do  the  offline  verification  step,  increasing  confidence  in  the 
correctness  of  the  models,  and  therefore  the  correctness  of  the  resultant  guarantees.  Based  on  the 
observations  above,  the  plan  is  to  develop  techniques  to  validate  design-time  models  during  system 
execution.  Specifically,  this  research  will  focus  effort  in  two  areas: 

1)  Research  and  develop  the  theory  and  techniques  to  monitor  indirect  models  of  software, 
initially  on  memory  access  patterns.  A  monitoring  module  will  check,  at  runtime  that  the 
observed  memory  access  pattern  matches  the  pattern  the  software  is  expected  to  produce, 
which  has  been  previously  shown  to  vary  significantly  across  programs.  This  means  that  a 
successful  attack  needs  to  not  only  exploit  the  system,  but  also  have  an  identical  memory 
access  pattern.  By  using  earlier  results  on  reshaping  memory  access,  the  proposed  approach 
can  create  a  system-specific  memory  profile,  meaning  that  an  attacker  needs  to  individually 
craft  each  attack  to  the  system  being  exploited;  the  same  exploit  code  cannot  be  used  to 
infect  multiple  systems. 
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2)  Monitor  assumptions  of  software  which  interacts  with  the  physical  world  by  using  hybrid 
automata  models.  By  periodically  sampling  the  system,  this  proposed  approach  can 
leverage  earlier  results  on  nonlinear  reachability  in  hybrid  automata  to  perform  a  time- 
bounded  reachability  computation  to  check  if  the  observed  state  is  reachable  in  the  model 
from  the  previously  sampled  state.  If  not,  the  assumed  model  of  the  system  is  incorrect, 
and  any  guarantees  proven  are  not  applicable  to  the  deployed  system. 
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4  -  Results  and  Discussion 


This  section  presents  the  results  of  the  investigation  into  fundamental  techniques  for  providing 
perpetual  model  validation,  that  is  validation  to  ensure  that  the  actual  behavior  of  the  system 
conforms  to  the  analysis  model,  raising  the  level  of  confidence  that  can  be  placed  in  the  results  of 
formal  analysis,  and  where  design-time  models  are  validated  continuously  during  runtime  using 
both  techniques  for  validation  of  the  software  model  and  validation  of  the  model  of  the  system 
interacting  with  the  physical  world. 

4.1  -  Year  One 

As  the  field  of  runtime  verification  has  extensively  focused  on  the  challenge  of  runtime  model 
validation  in  software  using  direct  models,  the  approach  employed  considered  using  indirect 
models  of  software  execution,  for  example  memory  access  patterns,  to  check  for  security 
intrusions.  Additional  research  was  performed  on  essential  problems  of  model  validation  for 
systems  which  contain  interactions  with  the  physical  world,  using  hybrid  automata  models.  Further 
details  are  presented  below  under  the  two  research  areas:  (1)  model  validation  through  improved 
analysis  of  hybrid  systems  reachability,  and  (2)  model  validation  using  analysis  of  indirect  models 
of  computation. 

4.1.1  -  Model  Validation  using  Hybrid  Systems  Reachability: 

A  ccomp  lish  men  ts: 

A  technique  to  deal  with  reducing  error  during  the  continuous  successor  operation  in  hybrid 
automata  reachability  algorithms  was  developed.  A  reduction  in  this  error  can  result  in  better 
accuracy  and  lower  computation  time.  For  perpetual  model  validation,  quick  computation  of  the 
continuous  successor  operation  is  necessary.  Results  are  presented  in  the  Workshop  on  Design, 
Modeling  and  Evaluation  of  Cyber  Physical  Systems  (CyPhy)  2014  [18]  and  the  ACM 
International  Conference  on  Hybrid  Systems:  Computation  and  Control  (HSCC)  2014  [19]. 
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Figure  2:  Wrapper  Ejfect  reduction  with  Pseudo-Invariants  [19]. 
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Figure  2  illustrates  at  a  high  level  what  the  application  of  this  approach  of  extracting  and  using 
Pseudo-invariants  can  achieve  when  computing  the  reachable  next  states  of  a  harmonic  oscillator. 
The  left  side  shows  the  reachability  approximated  through  the  standard  computational  means  while 
the  right  employs  pseudo-invariants.  The  comparative  area  of  the  resulting  approximation  is 
noticeably  reduced,  demonstrating  a  significant  decrease  in  the  wrapping  error  introduced  through 
over- approximation.  Further  details  are  presented  in  the  full  paper  [19]. 


A  key  enabling  technology  for  perpetual  model  validation,  the  ability  to  compute  hybrid  systems 
reachability  at  runtime,  was  developed.  The  first  Anytime  algorithm  for  reachability  was  proposed 
where  runtime  could  be  traded  off  for  accuracy.  Results  were  presented  at  the  2014  Safe  &  Secure 
Systems  &  Software  Symposium  (S5)  [20]  and  the  IEEE  Real-Time  Systems  Symposium  (RTSS) 
2014  [21]. 
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Figure  3:  Comparative  improvement  in  State-space  [21  ]. 


The  algorithm  allowed  for  a  large  improvement  in  the  manageable  state-spaces  for  small  run-times. 
Eigure  3  shows  the  results  of  the  work  published  in  [20,  21]  which  demonstrates  an  over  200% 
improvement  for  short  (~5-10ms)  run  times  and  continues  to  increase  the  longer  it  is  able  to  run. 

Additionally,  Professor  Taylor  Johnson’s  summer  faculty  project,  “Inferring  Physical  System 
Specifications  from  Embedded  Software  Tests,”  addressed  the  question  of  how  systems  can  be 
verified  without  a  formal  specification.  In  perpetual  model  validation,  some  form  of  model  is 
necessary.  By  using  the  developed  approach,  a  model  may  be  derived  from  initial  tests  or 
simulations,  which  will  then  be  formally  checked  at  runtime.  A  divergence  in  this  model  would 
imply  that  the  runtime  behavior  is  somehow  different  from  any  of  the  tests,  which  should  be 
investigated  by  a  system  designer. 

Further  Research 

The  real-time  reachability  results  need  to  be  evaluated  on  larger  systems  (more  continuous 
variables),  as  well  as  be  able  to  address  the  possibility  of  a  change  on  model  during  system 
execution.  LI  adaptive  control  allows  for  this  despite  limit  changes  in  a  model,  and  at  each  time 
step  produces  a  model  estimate  which  could  be  used  by  model  validation  approaches  and  will  be 
integrated. 

Real-time  reachability,  given  a  fixed  model,  can  check  if  the  runtime  sensor  readings  are  reachable 
from  the  previous  step  of  the  model,  which  is  similar  to  sensor-spoofing.  A  simple  remote- 
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controlled  car  and  indoor  localization  system  are  working  and  a  formal  model  of  the  vehicle  is 
being  identified.  After  this  step,  the  real-time  reachability  algorithm  will  be  used  during  runtime 
to  detect  when  the  localization  system  tries  to  guide  the  car  towards  an  obstacle. 

4.1.2  -  Model  Validation  using  Analysis  ol  Indirect  Models  ol  Computation 
Accomplishments: 

A  Cooperative  Research  and  Development  Agreement  (CRADA)  with  the  University  of  Illinois  at 
Urbana-Champaign  (UIUC)  was  established  to  build  on  this  basic  research  project  and  leverage 
their  earlier  research  on  the  Predictable  Execution  Model  (PREM)  for  real-time  system 
computation.  This  model  allows  for  the  division  of  the  computation  into  memory  and  execution 
phases.  Eight- PREM  is  UIUC  research  towards  automated  code  refactoring,  which  was  presented 
at  the  20th  IEEE  International  Conference  on  Embedded  and  Real-Time  Computing  Systems  and 
Applications  (RTCSA)  2014  [22] .  “Eight-PREM:  Automated  Software  Refactoring  for  Predictable 
Execution  on  COTS  Embedded  Systems,”  used  runtime  testing  to  automatically  determine  which 
memory  was  accessed  by  a  function  which  could  then  be  perfected  in  the  memory  phases. 

Further  Research 

The  current  automated  memory  refactoring  is  good;  however,  it  relies  on  observed  memory 
accesses.  Due  to  variations  in  run-time  addressing,  this  can,  in  some  cases  cause  SEGEAUETs. 
This  has  been  mitigated  by  capturing  the  SEGEAUETs  and  restoring  the  execution  state.  To 
achieve  better  efficiency,  application  of  formal  techniques  for  enumerating  the  possible  memory 
access  using  concolic  execution  should  be  implemented. 

The  MadT  tool,  developed  by  Marco  Cesati  at  the  University  of  Rome  (also  known  as  University 
of  Rome  Tor  Vergata),  should  be  investigated,  as  this  tool  uses  the  OS  to  exactly  detect  which 
memory  addresses  a  program  touches  and  it  is  expected  that  a  more  accurate  and  symbolic 
memory-address  identification  technique,  compared  with  PREM-light,  will  result. 

The  longer  term  step  would  be  to  create  patterns  of  memory  access  inside  of  specific  programs  to 
be  detected  by  the  OS  as  a  side-channel  as  part  of  perpetual  model  validation. 

4.2  -  Year  Two 

The  research  continued  to  focus  on  fundamental  issues  for  performing  validation  using  indirect 
models  of  software  execution,  for  example  the  behavior  of  physical  systems  which  are  coupled 
with  the  software  in  a  cyber-physical  system.  Mismatches  in  the  predicted  model  and  observed 
model  could  indicate  security  intrusions  or  other  critical  problems.  The  research  this  year  focused 
primarily  on  systems  formalized  using  hybrid  automata  models.  Roughly  speaking,  there  were  two 
research  thrusts:  (1)  providing  improved  techniques  to  perform  hybrid  systems  reachability 
analysis,  and  (2)  applying  the  developed  techniques  to  perform  system  validation. 
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4.2.1  -  IMPROVED  TECHNIQUES  EOR  HYBRID  SYSTEMS  REACHABILITY: 

Accomplishments: 

A  key  enabling  technology  for  perpetual  model  validation,  the  ability  to  compute  hybrid  systems 
reachability  at  runtime,  was  developed.  The  first  Anytime  algorithm  for  reachability  was  proposed 
where  runtime  could  be  traded  off  for  accuracy.  Results  were  published  at  RTSS  2014,  and  a 
journal  extension  of  this  work  which  evaluated  the  approach  on  a  nonlinear  model  as  well  as  its 
performance  on  embedded  hardware  was  accepted  for  publication  in  the  journal  Association  for 
Computing  Machinery  (ACM)  Transactions  on  Embedded  Computing  Systems  (TECS)  [23]. 

Professor  Taylor  Johnson’s  held  a  summer  faculty  research  position  and  his  project,  “Inferring 
Physical  System  Specifications  from  Embedded  Software  Tests,”  addressed  the  question  of  how 
systems  can  be  verified  without  a  formal  specification.  The  results  of  his  project  on  automatic 
model  synthesis  was  published  in  the  International  Conference  on  Cyber-Physical  Systems 
(ICCPS)  2015  [24,  25]. 


Figure  4:  Hynger  Overview  [24,  25], 


This  inference  process  is  carried  out  by  a  tool  referred  to  as  Hynger  (Hybrid  iNvariant  GEneratoR), 
overviewed  in  Eigure  4,  which  is  a  MATEAB  and  Java  based  tool  that  accepts  Simulink/Stateflow 
models  as  input  which  it  then  simulates  and  observes  in  order  to  generate  a  set  of  candidate 
invariants. 

A  tool  for  performing  hybrid  automaton  model  transformations,  Hyst,  was  developed  and 
published  as  a  Tools  paper  in  HSCC  2015  [26].  It  was  also  showcased  in  the  poster  /  demo  session 
of  CPS  week  2015.  Results  on  using  pseudo-invariants  from  last  year  were  automated  into  a  Hyst 
model  transformation  pass,  increasing  the  ease  of  applicability  of  earlier  results.  Eigure  5  illustrates 
the  flow  and  the  formats  supported  by  the  Hyst  tool.  It  was  designed  with  the  intent  of  being 
extensible,  through  the  intermediate  format,  to  potentially  support  other  sources  and  tools  in  the 
future. 
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Model 


Figure  5:  Hyst  Model  Transformation  Toolflow  [26], 


Further  Research 

The  Hyst  model  transformation  tool  will  continue  to  be  developed  with  other  passes,  such  as 
hybridization,  which  would  enable  nonlinear  hybrid  automata  to  be  analyzed  by  tools  which  only 
handle  linear  dynamics.  Furthermore,  the  automated  running  of  tools  and  automated  tool  chaining 
is  planned  to  make  analysis  both  more  scalable,  as  well  as  easier  to  apply  for  users  of  the  analysis 
tools. 

Models  of  cyber-physical  systems  often  include  tight  control  loops.  Reachability  tools  scale  poorly 
on  such  models,  due  to  a  large  number  of  discrete  transitions  which  occur  whenever  the  controller 
is  run.  However,  these  system  are  of  critical  importance  and  require  development  of  approaches 
which  permit  their  formal  analysis  by  creating  continuous  abstractions  of  the  periodically-actuated 
systems,  which  are  likely  to  scale  better  for  analysis. 

4.2.2  -  System  Validation  using  the  Developed  Techniques: 

Accomplishments: 

Runtime  results  of  reachability  can  be  used  to  provide  necessary  and  sufficient  conditions  for 
safety  to  a  distributed  cyber-physical  system  where  the  communication  is  unreliable  was 
demonstrated  in  the  ACM  TECS  [27]  journal  paper.  Additionally,  progress  guarantees  were 
possible,  under  the  assumption  that  communication  eventually  gets  through. 

Based  off  of  the  CRADA  established  with  the  University  of  Illinois  at  Urbana-Champaign,  the 
developed  real-time  reachability  algorithm  was  applied  towards  detecting  sensor  spoofing.  A 
remote-control  car  testbed  in  UIUC’s  lab  was  utilized  with  faked  sensor  readings  in  Matlab  to 
demonstrate  that  the  technique,  which  made  use  of  real-time  reachability,  detected  the  model 
violations. 

Further  Research 

Extend  current  results  by  utilizing  the  developed  real-time  reachability  technique  to  do  the  run¬ 
time  reachability  computation.  This  would  enable  the  analysis  of  more  complicated  dynamics 
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compared  with  the  existing  approach,  which  requires  solutions  to  the  differential  equations  be 
provided. 

4.3  -  Year  Three 

The  last  year  of  the  research  project  focused  primarily  on  ways  to  make  formal  methods  for  hybrid 
systems  reasoning  1)  more  automated  and  2)  applicable  to  a  broader  application  space. 

4.3.1  -  Improved  Automation  Techniques: 

In  terms  of  automation,  one  key  contribution  was  the  Hypy  tool.  Hypy  is  a  set  of  python  libraries 
which  automate  the  previously  manual  task  of  converting  models,  running  analysis  tools,  and 
interpreting  tool  results.  This  allows  powerful  high-level  analysis  approaches  where  tools  are  run 
multiple  times  and  models  and  tool  parameters  can  be  tuned  between  each  run.  The  applicability 
of  Hypy  in  automated  parameter  tuning,  automatic  model  modification  (via  the  previously  manual 
method  of  pseudo-invariants)  was  shown  to  improve  verification  accuracy,  and  iterative 
abstraction  refinement  methods  for  hybrid  systems.  The  audience  at  Applied  Verification  for 
Continuous  and  Hybrid  Systems  (ARCH)  2016  [28]  voted  Hypy  the  Best  Tool  Award.  Another 
contribution  for  automation  is  a  time-triggered  method  for  performing  static  hybridization  (multi- 
domain  linearization).  This  methodology  was  shown  to  scale  significantly  better  than  existing 
approaches  which  perform  a  state-based  division  of  the  variables.  The  computational  component 
for  this  work  was  written  with  Hypy,  and  this  paper  was  selected  for  the  Best  Repeatability 
Package  at  19th  ACM  International  Conference  on  Hybrid  Systems:  Computation  and  Control 
2016  [29]  from  approximately  20  entries. 

4.3.2  -  Broadened  Application  Space: 

While  the  formal  analysis  methods  researched  under  this  effort  have  traditionally  been  used  to  find 
bugs  in  existing  system  designs,  they  also  open  up  new  applications  which  expand  the  impact  of 
the  research  results.  Over  the  last  year,  a  journal  paper  in  Quantum  Information  Processing  that 
compares  traditional  parallel  solvers  for  graph-theoretic  NP-complete  problems  against  both 
quantum  computing  approaches  and  against  formal  methods  tools  (SMT  solvers)  has  been 
published.  The  SMT-based  verification  tools  in  this  case  solved  the  largest  problems  in  the  least 
time.  Additionally,  this  work  examined  conditions  on  when  a  cyber-physical  system  which 
requires  continuous  actuation  could  be  safely  restarted  (which  disconnects  the  controller  for  a 
bounded  amount  of  time).  This  serves  as  a  novel  robustness  mechanism,  by  keeping  the  system 
state  in  a  region  where  resetting  is  possible,  the  system  can  tolerate  faults  that  can  be  resolved 
through  a  full  system  restart  (which  generally  works  well  for  traditional,  software-only  systems). 
Further  details  may  be  found  in  the  presentation  at  the  21st  IEEE  Conference  on  Emerging 
Technologies  and  Eactory  Automation  (ETEA)  2016  [30].  The  research  effort  also  looked  at  the 
applications  of  hybrid  systems  reachability  in  order  to  detect  model  mismatches.  This  can  be  used 
for  quickly  finding  certain  kinds  of  security  intrusions,  as  well  as  detecting  when  the  physical 
model  of  a  system,  due  to  physical  deterioration  for  example,  no  longer  corresponds  to  the  model 
used  during  verification  time.  Einally,  the  research  looked  at  ways  to  combine  hybrid  systems 
reasoning  with  software  model  checking.  Through  the  use  of  a  new  object,  contract  automaton, 
results  from  the  two  domains  could  be  soundly  combined  to  enable  end-to-end  verification  for  a 
cyber-physical  system,  and  is  further  described  in  ACM  Special  Interest  Group  on  Embedded 
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Systems  (SIGBED)  International  Conference  on  Embedded  Software  (EMSOET)  2016  [31]. 
Further  Research 


Current  research  directions  include,  improving  Hyst  to  allow  for  model  specifications  that  include 
lookup  tables,  time  delays  and  stochastic  analysis;  detecting  deviations  from  predicted  memory 
access  profiles;  and  use  of  real-time  reachability  as  a  detection  mechanism  for  runtime  model 
mismatch. 
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5  -  Conclusions 


With  the  Air  Force’s  increasing  demand  for  mission-critical  functionality,  the  development  of 
correct  software  systems  is  fundamental  to  mission  assurance.  The  current  way  software  systems 
are  developed  and  maintained  often  produces  brittle,  out  of  date,  and  vulnerable  systems.  The 
systems  that  the  Air  Force  relies  on  need  to  be  predictable,  dependable  and  resilient.  The  software 
should  improve  mission  assurance  by  providing  a  fight  through  capability  that  might  degrade 
performance  but  still  accomplish  the  mission  objectives.  Modernization  of  software  systems  to 
patch  newly  discovered  flaws  or  hardware  updates  should  be  rapid  and  increase  the  robustness  of 
the  system,  not  make  the  system  more  brittle. 

General  purpose  software  development  has  demonstrated  that,  with  almost  certainty,  software  bugs 
will  be  present  in  the  resultant  code.  For  this  reason,  formal  methods  must  play  a  larger  role  in 
future  Air  Force  software  systems.  Safety  verification,  security-up-front  design,  and  other  early 
analysis  approaches  all  require  models  to  be  made  which  capture  the  expected  behavior  of  the 
system.  Perpetual  model  validation  enhances  and  complements  these  formal  methods  techniques, 
by  checking  that  the  expected  model  reflects,  as  far  as  one  can  tell,  the  behavior  of  the  deployed 
system.  If  a  violation  is  detected,  it  does  not  necessarily  mean  that  the  formal  properties  are  wrong 
and  will  be  violated,  only  that  their  proof  is  inadmissible.  This  in  itself  is  extremely  valuable 
information,  and  inferring  it  before  stressing  the  system  can  prevent  unexpected  violations  of 
safety  and  security.  The  proposed  research  will  increase  confidence  in  the  correctness  of  the 
models,  and  therefore  increase  confidence  in  the  correctness  of  the  resultant  formal  guarantees. 

Formal  design  of  the  software  cannot  only  benefit  from  perpetual  model  validation  using  indirect 
models,  but  as  the  space  and  air  domains  have  a  physical  aspect  to  them,  they  may  be  modeled 
using  hybrid  automata  in  order  to  validate  assumptions  the  software  makes  about  the  physical 
components,  and  the  assumptions  the  physical  components  make  about  the  software. 

This  research  contributes  toward  the  science  needed  to  build  safe  and  secure  systems.  It  is 
important  for  the  success  of  the  Air  Force  to  overcome  current  approaches  which  consider  security 
in  the  late  stages  of  the  development  of  a  system,  and  then  with  formulations  of  security  incapable 
of  providing  comprehensive  guarantees.  The  result  is  a  proliferation  of  attacks.  One  of  the 
Essential  Focus  Areas  for  the  Air  Force  is  Cyber  Resilience,  whereby  systems  would  be  able  to 
withstand  attacks.  A  superior  approach  is  to  prevent  many  of  those  attacks.  Research  into  perpetual 
model  validation  contributes  toward  that  goal. 

The  research  progressed  along  a  number  of  areas  centered  around  the  use  of  formal  methods  and 
hybrid  automata  models  for  the  run-time  analysis  and  verification  of  Cyber  Physical  Systems. 
Improved  methods  for  Run-Time  Assurance  have  been  tested,  frameworks  for  model 
transformation  and  generation  created,  and  Real-Time  scheduling  techniques  adapted  to 
incorporate  security  constraints. 

The  research  has  focused  on  checking  the  accuracy  of  continuous  models  at  runtime  through 
applying  formal  analysis  techniques  to  CPS  in  such  a  way  requiring  accurate  mathematical  models 
which  allows  for  the  detection  of  attacks  or  degraded  performance.  This  approach  thus  validates 
the  software  and  that  software’s  interaction  with  its  environment  through  hybrid  automata  based 
modeling. 
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With  the  Air  Force’s  increasing  demand  for  mission-critical  functionality,  the  development  of 
correct  software  systems  is  fundamental  to  mission  assurance.  The  current  way  software  systems 
are  developed  and  maintained  often  produces  brittle,  out  of  date,  and  vulnerable  systems.  The 
systems  that  the  Air  Force  relies  on  need  to  be  predictable,  dependable  and  resilient.  The  software 
should  improve  mission  assurance  by  providing  a  fight  through  capability  that  might  degrade 
performance  but  still  accomplish  the  mission  objectives.  Modernization  of  software  systems  to 
patch  newly  discovered  flaws  or  hardware  updates  should  be  rapid  and  increase  the  robustness  of 
the  system,  not  make  the  system  more  brittle. 

5.1  -  Way  Ahead 

Research  areas  for  both  new  designs  and  maintenance  actions  include  but  are  not  limited  to: 
Scalable  formal  methods  for  establishing  trust  of  resilient  systems;  Methodologies  for  complex 
software  design,  development,  analysis,  synthesis,  repair,  and  validation  and  verification;  System 
composition  analytics;  Formal  Models  of  composable  properties;  Trustworthy  architectures  for 
system  of  systems;  Modeling,  assessment,  and  vulnerability  analysis;  Assessment  and 
measurement  for  end-to-end  system  analysis;  Software  comprehension,  curation  and  diagnostics 
tools;  Methodologies  to  improve  understanding  of  software  including  the  reasons  behind  design 
choices;  ultimately  leading  to  Modular,  automated,  interoperable,  &  affordable  systems. 

Current  approaches  to  developing  resilient  systems  typically  lack  a  rigorous  assessment  of  trust. 
Resiliency  approaches  affect  system  change  in  an  attempt  to  fight  through  failures  and  attacks  but, 
at  most,  only  hand  wave  about  whether  or  not  these  changes  should  be  trusted.  How  do  we  know 
these  changes  will  lead  to  mission  success  or  mission  failure?  This  project  will  leverage  the 
Foundations  of  Trust  Program  and  maturing  resiliency  research  as  a  testbed  to  experiment  with 
assessing/reestablishing  trust  during  resilience  actions. 

There  are  two  thrusts  within  this  project.  First  is  the  extension  of  the  calculus  of  trust  to  encompass 
resiliency.  Second  is  extension  of  maturing  resiliency  research  prototypes  with  trust  and  critical 
experiment  and  demonstration  of  its  viability. 

Correctness,  vulnerabilities,  bugs  and  maintainability  (legacy  software)  have  been  key  DoD  and 
industry  challenges  for  as  long  as  software  have  been  developed,  deployed  and  maintained.  After 
years  of  investment  from  both  the  public  and  private  sectors,  we  are  now  at  a  crossroads  where  the 
fundamental  techniques  are  stabilizing  and  research  prototypes  are  being  demonstrated  on  real 
world  systems.  Take  for  example,  the  DARPA  High  Assurance  Cyber  Military  Systems  (HACMS) 
program  that  seeks  to  use  automated  tools  to  either  partially  or  fully  synthesize  the  control  software 
for  various  Unmanned  Platforms.  The  automatically  synthesized  code  has  correctness,  safety  and 
security  guarantees  with  respect  to  the  specification.  While  the  mathematical  proof  provides  a 
high  level  of  trust,  there  is  still  room  for  vulnerabilities  and  flaws  at  the  specification  level.  The 
time  and  expertise  required  to  use  the  formal  methods  tools  makes  it  necessary  for  any  repair  to  be 
accomplished  at  a  software  depot  or  prime  contractor.  The  technology  developed  under  the 
Foundation  of  Trust  program  and  others  promoting  the  use  of  formal  methods  would  benefit  from 
the  resiliency  research. 

The  later  thrust  will  research  two  existing  resiliency  research  prototypes  GenProg  and  Formally 
Generating  Adaptive  Security  Protocols.  GenProg  is  an  evolutionary  approach  that  is  inherently 
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resilient  due  to  its  ability  to  evolve  a  solution  but  is  inherently  less  trusted  since  it  uses  a  stochastic 
approach  to  fight  through  attacks.  Formally  Generating  Adaptive  Security  Protocols  is  a  formal 
approach  that  is  inherently  trusted  since  it  uses  formal  logic  to  synthesize  correct  by  construction 
code  but  is  inherently  less  resilient  since  it  is  currently  difficult  to  change  the  original  high  level 
specification  to  respond/adapt  to  successful  attacks.  This  project  will  explore  the  middle  ground 
between  evolutionary  and  formal  approaches  to  develop  trusted  and  resilient  systems. 

Resilient  software  approaches  that  rely  on  self-adapting  code  are  currently  not  trusted  by  the  end 
user.  While  this  program  will  explore  technological  solutions  to  this  problem,  it  will  also  have  to 
explore  the  human  issues  of  trusting  self-adapting  software.  To  accomplish  this  task  the  program 
will  have  multiple  directions.  One  important  area  that  will  be  pursued  is  the  readability  of  code 
generated  by  the  resiliency  tools.  Trust  can  be  gained  if  a  human  can  read  the  repair  code  and 
quickly  understand  and  confirm  the  modified  code  is  correct. 

Now  is  the  right  time  to  take  on  this  research  challenge.  Adaptive  response  is  widely  accepted  as 
a  necessity  to  survive  and  operate  through  attacks,  but  system  owners  and  operators  (warfighters) 
are  reluctant  to  use  this  new  technology  because  it  lacks  the  technological  advances  required  to 
earn  their  trust.  Leveraging  past  and  ongoing  research  in  trust  and  resilience  presents  an 
opportunity  for  developing  this  holistic  foundation  for  trusted  resilient  systems. 
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